Repository navigation
feat(routing): add opt-in latency SLO admission - #61
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | 6c67249 | Oct 09 2026, 05:06 AM |
6514d0c
into
integrate/gateway-waves-1-5
Summary
Adds opt-in latency-SLO admission. It predicts whether a request will finish within its deadline and can reject it, or reroute an automatic selection, before any provider handoff. With
LATENCY_SLO_MODEunset oroff(the default), admission, routing, responses and headers are byte-for-byte unchanged, and no observations are recorded.LATENCY_SLO_MODEisoff,rejectorreroute. Empty means off; a malformed value is off with one warning that leaves the value out.LATENCY_SLO_MIN_SAMPLES(default 20, 1–1000) andLATENCY_SLO_MAX_MODELS(default 128, 1–512).LATENCY_SLO_POLICY_JSON(default{}) is validated strictly. It names exact paths or auto model IDs underroutes, and authenticated key IDs underkeys, each withdeadline_ms(1–300000) and an optionalrequire_coverage. When several entries match, the smallest deadline and any coverage requirement win. A request that matches no entry uses ordinary policy.ttft_ms + output_tokens * 1000 / tokens_per_second. Output tokens come frommax_tokens/max_completion_tokens, or 1024 by default.prediction_unknown. That uses ordinary policy, unless the matching entry requires coverage; then the request gets 503latency_slo_unavailable.rejectmode returns 503latency_slo_predicted_missbefore any provider call. Both errors areretryable: false, carryCache-Control: no-store, and include the content-free prediction report.rerouteapplies only to automatic selections. It keeps already eligible alternatives with the same reviewed quality tier and a known prediction within the deadline. It never adds models or retries.Details are in
docs/latency-slo.md.Wiring
services/gateway_extensions.pyregisterslatency_slo_request_hookby name, right after the generation deadline hook and before the idempotency claim. It also adds a read-onlylatency_slo_candidate_policycollaborator for auto routes, which returnsNoneuntil it is supplied.services/intelligence_policy.py:select_candidatesapplies the policy to its final candidate list after session-tier selection. With the mode off it returns the same list.services/route_health.pyrecords observations (only when enabled) and passes automatic route ordering throughorder_auto_candidates. That returns the order unchanged unless the request has an active SLO decision.worker/latency-slo.mjsprovides the same settings and prediction for native Worker routes (prepareLatencySLO,recordLatencyObservation). It is wired into the native generation path with the Wave 6 integration.Shared registrations
tests/test_latency_slo_worker.mjsintest:worker.worker/latency-slo.mjsin the edge module inventory.LATENCY_SLO_MODE,LATENCY_SLO_MIN_SAMPLES,LATENCY_SLO_MAX_MODELSandLATENCY_SLO_POLICY_JSONin the Container env keys.tests/test_wave4_registrar.pyandtests/test_wave5_registrar.pynow expectlatency_slo_request_hookbetween the deadline and idempotency hooks.No D1 tables, Flask or Worker routes, or browser-readable headers.
Tests
tests/test_latency_slo.pyandtests/test_latency_slo_worker.mjscover:npm run test:worker: 1,649 passed.npm run test:integration: 7 passed.migrations:check(28 contiguous, additive migrations),test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 6. It is based on
integrate/gateway-waves-1-5, the Wave 5 integration branch (#60). The README line comes with the Wave 6 integration PR.